<html>
<head><meta charset="utf-8"><title>Eagerly parsing · t-compiler/rust-analyzer · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/index.html">t-compiler/rust-analyzer</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Eagerly.20parsing.html">Eagerly parsing</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="219767939"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Eagerly%20parsing/near/219767939" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> kev <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Eagerly.20parsing.html#219767939">(Dec 13 2020 at 16:15)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span>  Was wondering would there be any benefits, if the analyser eagerly parsed files that have been declared with mod?</p>
<p>For example; I start off at <a href="http://lib.rs">lib.rs</a> and it's content is:</p>
<div class="codehilite" data-code-language="Rust"><pre><span></span><code><span class="k">mod</span> <span class="nn">foo</span><span class="p">;</span><span class="w"></span>
<span class="k">mod</span> <span class="nn">foo2</span><span class="p">;</span><span class="w"></span>
<span class="k">mod</span> <span class="nn">foo3</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>So before analysis, I parse the files related to foo, foo2, and foo3 eagerly</p>



<a name="219767995"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Eagerly%20parsing/near/219767995" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> kev <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Eagerly.20parsing.html#219767995">(Dec 13 2020 at 16:16)</a>:</h4>
<p>The parser would not need to hold any state to do this I think, just check for <code>mod</code> declarations in the parsed file</p>



<a name="219768008"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Eagerly%20parsing/near/219768008" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Eagerly.20parsing.html#219768008">(Dec 13 2020 at 16:17)</a>:</h4>
<p>Parsing is done strictly on file basis</p>



<a name="219768050"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Eagerly%20parsing/near/219768050" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Eagerly.20parsing.html#219768050">(Dec 13 2020 at 16:18)</a>:</h4>
<p>parser doesn't even have access to a file system to load child files</p>



<a name="219768062"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Eagerly%20parsing/near/219768062" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Eagerly.20parsing.html#219768062">(Dec 13 2020 at 16:18)</a>:</h4>
<p>But, as long as you do <em>anything</em> with a crate, we parse all of it's files. Due to the nature of rust's name resolution algorithm, it's not really possible to postopne parsing within the crate</p>



<a name="219768164"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Eagerly%20parsing/near/219768164" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> kev <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Eagerly.20parsing.html#219768164">(Dec 13 2020 at 16:21)</a>:</h4>
<p>Hm I see, before I discard this idea, the parser can still have no access to the file system. A component who does would check the CST for mod declarations and call the parser on /foo.</p>
<p>The aim was to do parsing in parallel and have all necessary files parsed before resolution , but I think it introduces a bit more complexity for not that great a benefit</p>



<a name="219768301"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Eagerly%20parsing/near/219768301" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Eagerly.20parsing.html#219768301">(Dec 13 2020 at 16:24)</a>:</h4>
<p>That's a great idea actually, and something we'd want to implement</p>



<a name="219768307"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Eagerly%20parsing/near/219768307" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Eagerly.20parsing.html#219768307">(Dec 13 2020 at 16:24)</a>:</h4>
<p>although, it should be pushed a little bit further up the stack.</p>



<a name="219768330"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Eagerly%20parsing/near/219768330" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Eagerly.20parsing.html#219768330">(Dec 13 2020 at 16:25)</a>:</h4>
<p>Specifically, it's the <code>DefCollector</code> who should done parallelization. When it parsed the file and build an item tree out of it, it should spawn rayon jobs to parse and item-tree-ize <code>foo1</code>, <code>foo2</code> and <code>foo3</code>.</p>



<a name="219768353"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Eagerly%20parsing/near/219768353" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Eagerly.20parsing.html#219768353">(Dec 13 2020 at 16:26)</a>:</h4>
<p>But salsa currently doesn't support within-query parallelism</p>



<a name="219768424"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Eagerly%20parsing/near/219768424" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> kev <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Eagerly.20parsing.html#219768424">(Dec 13 2020 at 16:27)</a>:</h4>
<p><span class="user-mention silent" data-user-id="133169">matklad</span> <a href="#narrow/stream/185405-t-compiler.2Fwg-rls-2.2E0/topic/Eagerly.20parsing/near/219768330">said</a>:</p>
<blockquote>
<p>Specifically, it's the <code>DefCollector</code> who should done parallelization. When it parsed the file and build an item tree out of it, it should spawn rayon jobs to parse and item-tree-ize <code>foo1</code>, <code>foo2</code> and <code>foo3</code>.</p>
</blockquote>
<p>Oh that makes sense, thanks for clarifying and correcting</p>



<a name="219768492"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Eagerly%20parsing/near/219768492" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Eagerly.20parsing.html#219768492">(Dec 13 2020 at 16:29)</a>:</h4>
<p>Before we go down that route, we can also try parallelizing <code>prime_caches</code>. Last time I tried that I was hitting limitations of rayon, which could also cause issues when salsa supports this natively.</p>



<a name="219768552"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Eagerly%20parsing/near/219768552" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Eagerly.20parsing.html#219768552">(Dec 13 2020 at 16:30)</a>:</h4>
<p>Specifically rayon's behavior around propagating panics caused cancellation to take a lot longer than it should, resulting in poor UX and sporadic test failures</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>